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SECTION  1 


INTRODUCTION 


RESEARCH  OBJECTIVES 

d 

The  overall  objective  of  this  research  uas  to  develop  the  capabilitg 
to  predict  operator  performance  at  a  system  console.  To  be  useful 
in  industry,  this  prediction  should  be  available  prior  to  production 
of  a  prototype  or  simulator.  The  value  of  such  predictions 
is  the  opportunity  for  an  analyst  to  select  a  design  that  meets  system 
performance  requirements  while  optimizing  human  performance  —  and 
to  be  able  to  make  this  selection  before  hardware  and  software  design 
and  development  activities  commence.  For  this  study,  three  models 
were  examined  to  determine  how  each  could  contribute  to  the 
formulation  of  an  integrated  model  for  predicting  human  performance. 

An  additional  objective  was  the  capability  of  predicting  human  error 
detection  and  correction  probabilities.  Human  errors  that  are  not 
detected  and  corrected  by  the  operator  become  system  errors  which  may 
impact  system  reliability.  Human  errors  that  are  detected  and  cor¬ 
rected  remain  as  human  errors,  without  system  reliability  impact. 

The  act  of  detection  and  correction  of  errors,  however,  can  be  ex¬ 
pected  to  increase  system  performance  time.  For  a  task  that  has 
stringent  time  performance  requirements,  error  detection  and  correc¬ 
tion  may  not  be  possible  and  thus,  system  errors  may  result. 
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BACKGROUND 

A  computer  program  entitled  Human  Engineering  Computer  Aided  Design 
(HECAD)  uias  developed  by  AMRL.  This  program  uas  designed  to  pre¬ 
dict  operator  task  times  and  error  rates  at  operator  workstations. 

The  program  uses  Files  of  the  AIR  Data  Store  (Mungeri  Smithi  &  Payne> 
1962)  to  obtain  times  and  reliabilities  oF  control  activations  and  dis¬ 
play  readings;  and  methods-time-measur?ment  to  obtain  times  For  hand 
and  eye  transFers. 

Ford  Aerospace^  in  cooperation  with  AMRLj  conducted  a  laboratory 
test  oF  HECAD  to  validate  and  update  HECAD  predictions  (Goldbeck  Sc 
Charletc  1975).  Videotaped  hand  and  eye  movement  data  were  used 
to  modiFy  the  elements  oF  task  time.  Alsoc  a  distinction  was  made 
between  link  and  node  errors;  w^ere  going  to  the  wrong  control  or 
display  is  a  link  error<  and  mismanipulating  a  control  or  misreading 
a  display  is  a  node  error.  One  oF  the  limitations  oF  HECAD  is  that 
it  does  not  predict  error  detection  and  correction^  so  that  its  out¬ 
put  is  in  human  error  per  sei  and  not  the  eFFect  oF  human  error  on 
system  perFormance. 

Independent  oF  work  on  HECAD*  Warren  Teichner*  under  contract  with 
ONR  and  AFOSR  at  New  Mexico  State  University  (Teichneri  1974;  Teich- 
ncr  S(  Williams*  1979)*  had  developed  an  inFormation  transFer  theory 
oF  perFormance.  His  theory  proposed  at  least  Four  stages  oF  process¬ 
ing:  (1)  Stimulus  acquisition*  (2)  S-S  translations  (the  processes 

by  which  the  identiFied  stimulus  is  translated  to  a  new  code)*  (3) 
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S~R  translations  (translations  of  a  stimulus  coda  to  a  responsa  coda)< 
and  (4)  Response  executions.  The  S-S  and  S-R  translations  are  com¬ 
parable  to  a  HECAD  link.  Since  HECAD  link  types  were  used  for  HECAD 
prediction  of  link  errors^  Teichner's  translations  are  the  corres¬ 
ponding  predictors  for  the  Teichner  theory.  Like  the  HECAD  models 
the  Teichner  model  does  not  address  error  detection  and  correction. 


APPROACH 


The  relationship  between  the  HECAD  and  Teichner  models  is  impor¬ 
tant  to  arrive  at  an  integrated  prediction  model.  Points  of 
correspondence  must  be  established  so  that  this  integrated  pre¬ 
diction  model  can  be  developed  for  use.  A  third  component  of  the 
integrated  prediction  model  is  the  Information  Metric  which  is  an 
expression  of  central  processing  uncertainty.  In  order  to  begin 
developing  the  prediction  model*  two  components*  or  tasks*  must 
be  constructed.  The  first  req,uired  task  is  a  Data  Aci^uisition  task. 
The  major  purpose  of  this  task  is  to  identify  and  acquire  the  types 
of  data  necessary  to  establish  the  prediction  model.  The  second 
task  is  central  to  the  research  strategy  for  evaluating  the  model 
comprised  of  the  three  dimensions  listed  above.  This  second  task 
is  called  a  Validation  task.  The  purpose  of  the  Validation  task 
is  to  study  how  the  prediction  model*  based  on  data  from  the  Data 
Acquisition  task*  can  be  applied  and  tested  with  regard  to  human 
performance  predictions.  After  these  data  are  collected*  it  is 
then  possible  to  make  predictions  for  the  Validation  task.  Data  is 
then  collected  from  the  Validation  task*  so  that  the  performance 
predictions  and  the  prediction  model  can  be  evaluated. 
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SECTION  2 


METHOD 


SUBJECTS 


A  total  of  20  men  and  uiomen  betuieen  the  ages  of  19-33 
participated  as  subjects  in  the  present  studg.  All  subjects 
met  the  follouiing  criteria:  no  previous  experience  with  con¬ 
trol  panel  operationsi  no  physical  defects  which  would  impair 
their  performance  on  the  experimental  eiiuipmenti  20/20  normal 
vision  (corrected  or  uncorrected ) <  English  as  a  primary  con¬ 
versational  languagei  and  the  successful  completion  of  at 
least  one  semester  in  an  accredited  junior  college>  collegei 
or  university.  All  subjects  were  supplied  by  a  temporary 
employment  agency  and  were  paid  for  their  participation. 

APPARATUS 

In  order  to  present  the  experimental  display  and  task 
scenarios  to  the  subjects  the  following  system  configuration 
was  used  (See  Figure  1):  (1)  a  microcomputer  with  standard 

peripheralsi  (2)  a  display  generator  with  interfaces  to  the 
computer  system>  (3)  a  cathode-ray  tube  (CRT)  display  monitori 
and  (4)  a  touch  entry  device.  A  brief  description  of  these 
hardware  items  follows. 
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Figure  1.  System  Configuration 
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Computer.  A  16-bit  DEC  LSI-11  digital  microcomputer 
and  line  printer  luere  used  to  generate  both  display  formats  as 
uiell  as  to  record  and  store  subject  time  and  error  responses. 


Disolau  generator.  The  display  generator#  developed 
by  FACC>  is  a  high  resolution#  raster  scan  display  unit  capable 
of  generating  characters  in  a  7  x  9  pixel  format  (9x9  font). 

In  addition#  the  display  generator  uas  capable  of  presenting 
lines  to  define  a  boundary  around  each  switch  area  used  for  re¬ 
sponses  on  the  touch-entry  CRT.  When  this  switch  area#  or  matrix 
box#  was  selected  by  the  user#  the  inside  of  the  box  went  to  reverse 
video  to  indicate  that  a  specific  area  had  been  selected.  Only 
one  switch  area  selection  could  be  activated  or  be  in  reverse  video 
at  any  given  time. 


Disolau  monitor.  A  17"  Conrac  CRT#  (Model  No.  RGB  17/N# 
P4  phosphor)#  was  used  to  display  all  stimuli.  This  CRT  is  a 
high  resolution  (721  X  826  lines  by  pixels)  black  and  white 
display  with  a  character  size  of  7  x  9  pixels.  The  CRT  has 
a  character  brightness  of  50  ft.  L  and  a  contrast  ratio  of  4: 1. 


Touch  entru  device.  The  touch  entry  device#  manufactured 
by  Carroll  Manufacturing#  was  mounted  around  the  periphery  in  front 
of  the  CRT  surface.  A  total  of  88  LED  emitters  and  phototransistor 
detectors  were  mounted  1/4"  apart  to  produce  a  beam  matrix  of  48 
(horizontal)  x  40  (vertical).  Thus,  an  11  3/4"  x  9  3/4"  area  of  the 
CRT  was  capable  of  acting  as  a  touch  entry  switch  matrix#  with  the 
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intersoctiort  of  tha  horizontal  and  vortical  beams  constituting  a 
touch  point.  Interruption  of  an  infra-red  horizontal  or  vertical 
LED  beam  by  the  operator's  finger  resulted  in  sending  a  specific 
CRT  coordinate  to  a  temporary  storage  register  located  in  the 
interface.  This  coordinate  uas  then  checked  by  the  software  to 
insure  that  it  was  a  valid  point,  i. e. >  located  within  a  particu¬ 
lar  box  of  the  display.  Once  validated,  the  coordinate  was  pro- 
cessed  and  transmitted  to  the  main  computer  for  storage  on  a  floppy 
disk.  Operator  feedback  of  switch  activation  was  provided  by  a 
15  msec  tone  and  the  selected  box  area  going  to  reverse  video  when 
the  CRT  screen  was  touched. 


Visual  disolau.  All  subjects  were  seated  directly  in  front 
of  the  CRT  so  that  the  center  of  the  display  matrix  was  approximately 
20  deg  below  the  line  of  sight  and  all  matrix  boxes  could  be  easily 
touched  by  the  operator.  The  subject's  console  was  positioned  in 
such  a  way  that  the  experimenter  could  observe  all  actions  through 
a  one-way  mirror. 

The  visual  display  was  composed  of  four  elements;  namely,  a 
matrix  area,  a  variable  status  field,  a  time  out  error  box.  and  a 
switch  error  box.  The  matrix  portion  of  the  display  area  consisted 
of  36  boxes  arranged  in  a  6  x  6  (row  x  column)  format  (See  Figure 
2).  Each  box  or  matrix  area  was  covered  by  three  horizontal  and 
three  vertical  active  beams.  The  intersection  of  these  beams  con¬ 
stituted  a  valid  touch  point,  thus  sending  a  coordinate  to  the  com¬ 
puter  when  the  inner  portion  of  the  box  was  touched.  Each  matrix 
area  box  was  separated  by  two  non-active  beams.  If  the  subject 
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Matrix  display 
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touched  any  part  of  the  display  outside  of  the  matrix  boxes>  that 
coordinate  was  considered  invalid  and  was  not  sent  to  the  computer. 

Thirty  four  of  the  matrix  boxes  corresponded  to  the  control 
actions  necessary  to  complete  both  tasks.  The  layout  for  the  matrix 
area  was  designed  in  such  a  way  that  control  responses  that  were 
correct  for  the  given  state  of  a  display  item  were  not  located  ad¬ 
jacent  to  the  correct  control  response  for  a  different  state  of 
the  same  display  item.  This  design  concept  was  followed  so  that  an 
accidental  activation  of  a  matrix  box  could  be  identified  as  a  node 
error  rather  than  the  more  cognitive  link  error. 

Located  to  the  right  of  the  matrix  area  was  a  21  item  variable 
display  status  field.  At  the  start  of  each  new  task  segments  each 
item  in  the  status  field  had  a  dashed  line  opposite  it.  The  dashed 
lines  were  eventually  replaced  by  the  words  yes>  no-go>  or  marginal 
when  a  corresponding  matrix  area  was  activated*  thus  giving  the 
operator  a  current  and  up-to-date  status  of  the  system.  Unlike  the 
matrix  area  boxes*  the  status  field  display  items  were  arranged 
sei^uential ly  from  top  to  bottom. 

The  third  element  of  the  visual  display  was  a  time  out  error 
box.  This  was  located  above  the  matrix  area  and  status  field.  The 
time  out  error  light  acted  as  a  prompt  to  get  subjects  to  make  a  matrix 
selection  when  too  much  time  was  being  taken  between  two  consecutive 
actions.  Subjects  were  given  a  total  of  20  sec  between  correct 
matrix  selections.  After  10  sec  had  elapsed  between  actions  the  time 
out  error  box  would  briefly  go  to  reverse  video.  This  was  basically 
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to  give  tha  subjact  a  cue  as  to  how  much  tima  had  alaspad.  If  too 
much  tima  was  still  taken«  and  a  correct  matrix  box  was  not  salactad> 
the  time  out  error  box  would  go  to  reverse  video  and  the  matrix  area 
would  lock  upi  meaning  that  no  matrix  selections  could  be  activated 
or  sent  to  the  computer.  Mhen  this  occurredi  the  experimenter  would 
give  the  subject  the  next  correct  action  and  restart  the  program. 

The  final  element  of  the  visual  display  was  the  switch  error 
box.  This  was  located  below  the  matrix  area  and  status  field.  The 
switch  error  box  served  as  a  feedback  cue  to  inform  the  subject 
that  an  error  had  been  made.  This  part  of  the  display  would  go 
to  reverse  video  only  when  two  consecutive  incorrect  matrix  box 
areas  were  selected.  The  switch  error  light  remained  in  reverse  video 
until  a  correct  matrix  selection  was  made. 

Error  keus.  Mithin  the  matrix  area  two  boxes  were  non¬ 
task  related;  namely<  the  ERROR  FEEDBACK  key  and  the  ERROR  IN¬ 
TUITIVE  key.  These  boxes  were  located  in  the  lower  left  and  right 
portion  of  the  matrix  display  area.  Subjects  were  instructed  that 
they  were  to  use  one  of  the  error  keys  whenever  an  incorrect  matrix 
switch  was  activated.  The  instructions  informed  the  subject  to  use  the 
ERROR  FEEDBACK  key  when  one  of  the  following  conditions  occurred: 
the  switch  error  light  went  to  reverse  videoi  the  equipment  locked 
up  and  the  experimenter  gave  the  next  correct  response#  or  the 
subject  made  an  incorrect  matrix  activation  and  did  not  realize  it  until 
he  obtained  additional  information  from  the  status  field.  Similarly# 
subjects  were  instructed  to  use  the  ERROR  INTUITIVE  key  for  the 
following  conditions;  the  wrong  matrix  switch  was  accidently  acti- 
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vated<  or  the  subject  immediately  realized  that  an  error  was  made. 

In  order  to  ensure  that  all  subjects  fully  understood  the 
proper  use  of  each  error  key<  a  list  of  possible  error  conditions 
was  given  to  each  subject.  Subjects  were  asked  to  categorize>  by 
means  of  a  true-false  q.uiz<  each  error  condition  by  the  error  key 
they  would  use  for  that  given  situation.  Immediately  after  com¬ 
pleting  this  categorization,  the  experimenter  went  over  each  error 
condition  again  with  the  subject  to  ensure  that  the  distinction  be¬ 
tween  the  two  keys  and  when  to  use  them  was  clear. 


PROCEDURE 


The  training  procedure  for  both  tasks  was  twofold.  The 
first  part  of  the  training  consisted  of  the  actual  learning  and 
memorization  of  the  task  seq.uences.  This  was  accomplished  through 
the  use  of  detailed  tapes  outlining  the  correct  actions  and  manipu¬ 
lations  necessary  to  perform  the  specific  task.  The  second  part 
of  the  training  procedure  provided  the  subjects  with  the  oppor¬ 
tunity  for  practice  and  memorization  of  the  task  while  becoming 
familiar  with  the  actual  experimental  eq.uipment.  Further  elabora¬ 
tion  on  the  training  procedure  follows. 

Training  tapes.  The  training  tapes  for  each  task  were 
divided  into  three  segments,  namely,  a  go  condition,  a  no-go  con¬ 
dition.  and  a  marginal  condition.  Two  tapes  accompanied  each  con¬ 
dition.  The  first  tape  presented  each  subject  with  a  detailed 
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step  by  step  account  of  the  task  to  be  performed/  as  uell  as  the 
rationale  for  taking  each  action.  The  second  tape  listed  only  the 
control  and  display  items  the  subject  needed  to  activate  or  check 
for  each  condition;  all  rationale  uas  omitted.  In  the  construction 
of  these  scenarios/  an  attempt  was  made  to  cast  both  tasks  in  a 
logical  format  to  expedite  the  memorization  of  the  task  actions. 

The  go  condition  description  outlined  the  dominant  path  for 
the  Data  Acquisition  task  as  well  as  for  the  S/alidation  task.  This 
condition  was  comprised  of  28  total  actions  consisting  of  matrix 
switches  to  activate  or  status  field  display  areas  to  check. 

The  no-go  and  marginal  conditions  were  both  based  on  the 
dominant  path  sequence  given  in  the  go  condition  scenario;  but  also 
contained  corrective  or  branch  actions.  The  no-go  and  marginal  con¬ 
ditions  had  a  total  of  82  and  65  matrix  activations  or  display 
checks  for  the  operator  to  memorize. 

Two  different  task  scenarios  were  used  in  the  present  study. 
The  first  task  run  was  the  Data  Acquisition  task.  The  subject's 
task  for  this  portion  of  the  study  was  to  construct  and  test  a  printed 
circuit  board  (PCB).  The  major  actions  for  this  task  required  the 
subject  to  print  circuits/  install  capacitors  and  resistors/  as  well 
as  to  test  for  acceptable  current  output  levels.  For  the  Validation 
task/  the  subject's  task  was  to  launch/  track/  command/  and  receive 
telemetry  from  a  satellite  by  manipulating  matrix  areas  and  reading 
status  field  display  areas.  Both  tasks/  construction  of  the  RGBs 
and  satellite  tracking/  used  the  same  dominant  path  and  branch  se¬ 


ta 


1 


Ford  Aorospaco  & 
Communieatlona  Corporation 

nuances.  Tha  validation  portion  of  this  study  uias  primarily  usad 
to  ansura  that  tha  performance  scores  and  errors  obtained  uere  due 
to  the  HECAO  link  types*  the  Teichner  translation  classifications* 
or  the  Information  Metric  value  and  not  due  to  scenario  differences. 


Training  session.  Training  for  both  the  Data  Acquisition 
task  and  the  Validation  task  required  approximately  6  hours.  All 
training  was  accomplished  through  the  utilization  of  training  tapes 
and  a  pictorial  mock-up  of  the  actual  display.  Each  session  in¬ 
cluded  an  explanation  of  the  control  (matrix)  and  display  (status 
field)  items*  followed  by  a  detailed  description  or  scenario  of  the 
specific  task  to  be  performed. 


Prior  to  the  training  session  a  brief  introduction  was  given 
to  each  subject.  During  this  time  the  mock-up  was  presented  in 
order  to  familiarize  the  subject  with  the  general  hardware  con¬ 
figuration  and  the  matrix  and  status  field  names.  In  addition  to 
the  hardware  familiarization*  all  subjects  were  given  a  general 
overview  of  the  purpose  of  the  experiment*  a  briefing  on  the  nature 
of  the  task*  and  a  true-false  quiz  on  the  use  of  the  error  keys. 

This  quiz  was  used  to  ensure  that  all  subjects  were  aware  of  the 
error  situations  that  went  along  with  either  the  error  feedback  or 
error  intuitive  keys. 


The  training  sessions  for  each  condition  consisted  of  two 
repetitions  for  each  segment.  For  the  first  repetition*  the  subject 
listened  to  the  tape  while  the  experimenter  pointed  to  the  corres¬ 
ponding  matrix  and  status  field  items.  This  was  done  in  order  to 
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familiarize  the  subject  uith  the  correct  item  locations  as  well  as 
to  eliminate  unnecessary  scanning  of  the  display  by  the  subject. 

The  phrase  "your  next  action"  served  as  a  cue  to  signify  that  the 
next  action  was  being  introduced.  At  the  end  of  a  given  se(iuence  of 
actionsi  the  tape  was  stopped  and  the  subject  was  asked  to  repeat  the 
exact  setiuence  of  actions  while  pointing  to  the  correct  matrix  and 
status  field  items.  The  subject  was  not  allowed  to  continue  to  the 
next  action  until  he  had  performed  the  previous  one  correctlyi  either 
with  or  without  the  experimenter's  assistance. 

For  both  tasks<  Acquisition  and  Validationi  the  same  pro¬ 
cedure  was  used  for  all  three  conditions.  Time  and  error  scores 
were  recorded  during  the  entire  training  session;  but  only  for  the 
second  repetition  of  each  short  sequence  of  actions.  These  time 
and  error  scores  were  only  used  to  establish  a  baseline  evaluation 
for  a  subject's  training  performance.  If  at  this  time  the  experi¬ 
menter  estimated  that  a  subject's  performance  would  not  reach  asymp¬ 
tote  in  the  time  available^  the  training  session  was  terminated  and 
the  subject  was  dismissed.  For  the  Acquisition  task/  a  total  of 
seven  out  of  nineteen  individuals  were  terminated  due  to  poor  train¬ 
ing  performance  or  cancellations.  For  the  Validation  task  there 
were  only  two  cancellations  out  of  ten.  Short  rest  periods/  as  well 
as  an  hour  lunch  break/  were  provided  throughout  the  training  session. 

Practice  trials.  Three  practice  trials  were  given  to  each 
subject  prior  to  the  actual  data  collection  trials.  The  practice  trials 
served  as  a  means  to  acquaint  the  subject  with  the  actual  equipment 
that  they  would  be  using/  as  well  as  to  give  them  an  example  of  how 
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the  task  sc^uancas  ware  put  together.  During  the  first  two  practice 
trials  the  experimenter  was  seated  next  to  the  subject  in  the  experi¬ 
mental  room.  By  sitting  next  to  the  subject*  the  experimenter  was 
able  to  observe  how  the  subject  interacted  with  the  equipment*  i.  e.  * 
were  they  touching  the  CRT  correctly.  Also  the  experimenter  could 
prompt  them*  if  necessary*  as  to  what  the  next  correct  control  action 
should  be.  For  the  third  practice  trial*  the  subject  was  seated 
alone  in  the  experimental  room  and  all  communications  were  via  an 
intercom.  During  this  time  the  experimenter  was  seated  in  an  adjoin¬ 
ing  room  with  the  computer  system.  In  addition  to  being  able  to 
observe  the  subject's  performance  through  a  one-way  mirror*  a  second 
CRT  was  available  for  viewing  the  matrix  e'rea  and  status  field.  This 
procedure  was  continued  throughout  the  entire  experimental  data 
collection  trials. 


Experimental  session.  The  data  collection  trials  required 
approximately  6  hrs  per  subject.  A  total  of  37  data  collection 
trials  were  given  to  each  subject.  The  subject's  task  for  the  Data 
Acquisition  portion  of  the  study  was  to  construct  and  test  a  printed 
circuit  board  via  the  touch-entry  CRT.  The  major  actions  for  this 
task  required  the  subject  to  print  circuits*  install  capacitors  and 
resistors*  as  well  as  to  test  for  acceptable  current  output  levels. 
For  this  task  a  trial  consisted  of  constructing  and  testing  four 
printed  circuit  boards  for  a  total  of  314  actions.  For  the  Valida¬ 
tion  portion  of  the  study*  the  subject's  task  was  to  launch*  track* 
command*  and  receive  telemetry  from  a  satellite.  As  with  the  Ac¬ 
quisition  task*  a  Validation  task  trial  consisted  of  tracking  and 
commanding  four  separate  vehicles  for  a  total  of  314  actions. 
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Subjects  mere  not  interrupted  or  prompted  during  data  collec¬ 
tion  unless  they  failed  to  respond  correctly  uithin  the  prescribed 
period  of  time.  When  this  time  limit  of  20  sec  uas  exceededi  the 
touch  entry  device  mould  lock  up>  i. e. <  no  matrix  smitch  inputs 
mere  sent  to  the  computer.  At  this  time  the  experimenter  mould 
intervene)  tell  the  subject  mhat  the  next  correct  action  should  bei 
and  re-initiate  the  computer  program  to  continue  the  trial. 


At  the  end  of  every  four  boards  or  vehicles  the  computer 
system  mould  lock  up  signifying  the  end  of  a  trial.  At  this  time 
the  experimenter  mould  ask  the  subject  if  he  had  any  comments  or 
problems  mith  the  eq.uipment  or  task.  If  none/  or  mhen  the  problem 
mas  solved)  the  program  mas  re-initiated  and  a  nem  trial  mas  started. 
Rest  periods  mere  given  at  appropriate  intervals  throughout  the 
data  collection  trials.  Time  data)  and  all  matrix  smitch  selec¬ 
tions)  mhether  correct  or  incorrect)  mere  recorded  by  the  computer 
for  all  trials. 


Preliminary  test  runs.  A  total  of  five  pilot  subjects  mere 
trained  and  tested  on  the  Data  Acquisition  task.  The  data  collected 
mere  used  to  validate  the  training  procedures)  to  establish  a  time 
baseline  for  training  and  testing)  to  establish  asymptote  criteria) 
and  to  determine  the  required  number  of  subjects.  The  preliminary 
data  collected  mere  not  used  in  the  final  data  analyses  for  either 
task.  For  the  Validation  task)  it  mas  not  deemed  necessary  to 
train  or  test  pilot  subjects  because  the  baseline  data  collected 
from  the  Acquisition  task  could  be  generalized  to  the  Validation  task. 
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DebrieFino  of  sub  lects.  Immediateltj  follouing  the  last  data 
trial<  all  subjects  uiere  given  a  (questionnaire  to  fill  out.  The 
questionnaire  uas  an  attempt  to  get  feedback  from  the  subjects  on  both 
the  training  and  data  collection  portions  of  the  stud(j.  Subjects 
mere  also  asked  if  theij  experienced  any  difficulty  learning  the  taski 
difficulty  using  the  correct  error  keys»  or  problems  interacting 
(Kith  the  equipment. 
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SECTION  3 


RESULTS 


PRE-EXPERIMENTAL  ANALYSIS 

The  follouing  findings  mere  derived  before  ang  experimental  data  uiere 
collected  for  the  present  study. 

Teichner  and  HECAD  Models 

Initial  efforts  uiere  devoted  to  obtaining  a  thorough  understanding 
of  the  theory  behindi  and  implementation  of/  the  Teichner  Model. 

All  available  documentation  of  the  Teichner  translation  implementation 
was  concerned  with  operator  task  performance  during  the  early  stage 
of  the  learning  curve.  Since  many  of  the  S-S  translations  that 
occur  in  preasymptotic  performance  drop  out  when  the  operator  reaches 
asymptote/  the  type  and  number  of  S-S  translations  indicated  in  this 
documentation  is  not  wholly  applicable  to  our  study  contract#  where 
only  asymptotic  performance  is  analyzed.  Another  consideration  is 
the  somewhat  (admittedly#  according  to  Teichner)  subjective  tech¬ 
nique  of  applying  an  S-S  translation  to  a  task  link.  Ue  attempted 
to  solve  these  difficulties  by  assuming  that  only  one  S-S  translation 
remained  for  each  task  link#  and  then  describing  in  operational 
terms  which  type  of  S-S  translation  must  occur  at  that  task  link 
based  upon  network  configuration. 
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This  assumption  is  based  upon  (1)  our  judgement  that  compression 
links  mill  drop  out  at  asymptote  and  (2)  the  fact  that  the  5-5  crea¬ 
tions  that  appear  in  Teichner's  work  are  due  to  multi-use  keys<  which 
are  not  used  in  our  study.  A  final  point  in  this  discussion  is 
the  fact  that  in  all  cases>  in  both  Teichner's  examples  and  our  task 
network,  the  5-R  translation  is  the  same  and  therefore  a  constant. 

Examination  of  the  relationships  between  models  has  shown  that  HECAD 
links  and  Teichner  translations  are  comparable  units  of  description, 
even  though  there  is  not  a  high  correlation  between  the  values  of  each. 
The  HECAD  link  types  and  Teichner  translation  types  do  not  track  one 
another  in  the  sense  that  knowing  link  type  does  not  specify  the  type 
of  translation.  However,  there  is  a  match  up  between  the  task  des¬ 
criptors  such  as  link  and  translation.  This  match  up  is  shown  in 
Figure  3. 

The  HECAD  task  action  is  shown  in  the  upper  half  of  the  figure,  and 
the  corresponding  Teichner  T-task  is  shown  in  the  lower  half.  It 
can  be  seen  that  the  HECAD  stimulus  and  response  nodes  correspond 
with  the  Teichner  stimulus  act^uisition  and  R-execution.  res¬ 
pectively.  and  that  the  HECAD  response  link  matches  up  with  the 
Teichner  5-5  and  5-R  translations.  Next  to  be  considered  is  what 
happens  with  error  responses.  The  HECAD  link  error  corresponds  with 
the  Teichner  translation  errors.  Both  involve  a  detection 
stimulus.  The  HECAD  correction  link  returns  to  the  correct  response 
node,  while  one  or  both  of  the  Teichner  translations  return  to  the 
correct  R-execution. 
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DETECTION  . 
•1'  STIMULUS  ! 

I  acquisition! 


I - 1  [ - ( 

I  S-S  1 _ ^  s-n 

ItranslationJ^  “translation! 

I - 1  I - 1 


Figure  3.  Correspondence  of  HECAD  and  TEICHNER  Models 
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Existing  Reliabilitu  Data 

The  next  step  in  our  uork  uas  to  determine  luhat/  if  any/  q.uantitative 
relationships  uere  shared  by  the  HECAD  and  Teichner  Models#  as  well 
as  the  Information  Metric.  To  accomplish  this  we  used  (1)  HECAD 
link  type  reliability  data  from  Task  0  of  a  previous  Operator  Per¬ 
formance  Study  (Goldbeck  &  Char let# 1975 > #  (2>  derived  performance 

reliabilities  for  each  of  the  three  types  of  Teichner  translations 
(S-S  conservation#  S-S  creation.  S-S  classification)#  and  (3)  derived 
performance  reliabilities  for  the  amount  of  information  processed. 

The  amount  of  information  processed  was  calculated  by  the  formula; 

n 


Where , 


H  =  expression  of  central  processing  uncertainty, 
o 

n  a  the  number  of  alternatives, 

Pj  =  the  probability  of  the  1th  alternative. 

The  resultant  values  of  this  Information  Metric  were  divided  into 
three  link  types:  high  information  content#  medium  information  con¬ 

tent#  and  low  information  content#  based  upon  an  eq.ual  sample  size 
in  each  group.  These  three  link  characteristics  were  then  combined  into 
a  3-dimensional  matrix  <see  Table  1)#  with  reliabilities  computed 
for  each  cell. 
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TABLE  1 


TASK  B  RELIABILITIES 


INFO 

METRIC 

HECAO 

LINK 

TYPE 

TEICHNER  TRANSLATION  TYPE 

Great i on 

Classification 

Rechec  k 

1.  0000/4320 

^  _ 

Dorn  Path 

. 9996/5120 

— 

— 

Lou 

Mid-branch 

.  9971/2080 

— 

— 

Start/return 

1.  0000/160 

— 

— 

V  1-link 

'  ■ 

Rechec  k 

_ „ 

1. 0000/640 

1.  0000/320 

Dorn  Path 

— 

1. 0000/2560 

. 9990/6720 

Medium 

Mid-branch 

— 

1. 0000/160 

.  9981/2000 

Start/return 

.  9854/480 

.  9953/3840 

1-link 

1. 0000/160 

.  9906/960 

Recheck 

1. 0000/3040 

Dom  Path 

— 

.  9981/1600 

.  9984/1280 

High 

Mid-branch 

— 

. 9988/1600 

.  9984/640 

Start/return 

— 

. 9938/1440 

.  9921/2400 

1-link 

— 

. 9979/480 

.  9292/480 

All  values  given  are  Rel iab i 1 i ty/Samp le  Size 
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As  can  be  soon  from  Table  1<  only  23  of  the  possible  43  cells  were 
filled.  The  unused  cells  were  due  to:  (1)  the  division  of  the  In¬ 
formation  Metric  into  three  e^ual  samples  was  such  that  low  informa¬ 
tion  content  equated  to  tero  information  processed.  Creation  and 
classif ication  types  of  S-S  translations)  which  by  definition  imply 
multiple  outputs  from  the  initial  node  of  the  link/  and  therefore) 
information  to  be  processed)  cannot  occur  in  a  zero  information  con¬ 
tent  cell)  (2)  conversely)  cells  that  contain  either  medium  or  high 
amounts  of  information  processed  preclude  use  of  an  S-S  conservation) 
since  by  definition  S-S  conservations  re(iuire  zero  amounts  of  in¬ 
formation)  and  (3)  for  the  same  reason)  a  HECAD  one-link  branch  cannot 
be  a  S-S  conservation  because  a  one-link  branch)  by  definition)  im¬ 
plies  that  the  initial  node  has  multiple  outputs. 

Because  Task  B  had  been  designed  without  T-Task  translations  or  In¬ 
formation  Metric  reliabilities  in  mind)  and  because  of  the  many  re¬ 
sultant  low  cell  f req.uenc ieS)  the  results  can  only  be  taken  as  suggestive. 
Nevertheless)  in  the  nine  contrasts  of  the  creation  and  classifica¬ 
tion  translations)  the  creation  was  more  or  ei^ually  reliable  than 
the  classification  translation  in  seven  of  the  nine  contrasts.  Also) 
in  the  nine  contrasts  of  the  medium  amount  of  information  with  the 
high  amount)  the  medium  amount  of  information  cells  were  more  or 
e(iually  reliable  than  the  high  information  cells  in  all  but  two  of 
the  contrasts.  In  the  nineteen  contrasts  of  adjacent  HECAD  link 
types  that  were  ranked  according  to  reliability  expectation)  there 
were  only  four  reversals.  Most  suggestive  of  a  relationship  between 
the  HECAD  and  Teichner  models  was  the  tendency  for  HECAD  link  types 


23 


Ford  Aoroapaco  & 
Communicationa  Corporation 

to  be  associated  with  larger  performance  differences  when  coupled 
with  a  classification  than  with  a  creation  or  conservation.  These 
encouraging  results  justified  the  development  of  tasks  that  specifi¬ 
cally  test  the  integrated  model. 

Task  Q  marginal  reliabilities  of  the  integrated  model>  giving  mean 
values  for  the  HECAOi  Teichneri  and  Information  Metric  variablesi 
are  shown  in  Table  2.  These  values  establish  the  ran*-  ordering  that 
is  tentatively  predicted  for  the  Acq,uisition  task.  Results  for  the 
Acquisition  task  will  provide  a  more  formal  basis  of  prediction  for 
the  Validation  task. 


Table  2 

Task  B  Marginal  Reliabilities 


HECAD  LINK 

TYPE 

Rechec  k 

1. 0000/8320 

Dorn  Path 

. 9999/17280 

Mid-branch 

.  9980/6560 

Start/return 

.  9936/8320 

1-link 

. 9788/2080 

TEICHNER 

TRANSLATION 

TYPE 

Conservation 

9993/11680 

Creation 

9982/12160 

•M  LI  lliM 

INFORMATION 

METRIC 

Low 

.  9993/11680 

Med ium 

.  9975/17920 

isTmi 

_ ■  9945/12960 

All  values  given  are  Rel iab i lity /Samp le  Size 
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Data  Collaction  Strateou.  Error  detaction  and  correction  predic¬ 
tion  algorithms  uiill  be  examined  in  a  manner  similar  to  performance 
reliabilities!  because  me  hypothesize  that  detection  and  correction 
probabilities  are  a  function  of  HECAD  link  typei  T-task  translation 
typei  and  the  amount  of  information  processed.  The  error  H^tection 
and  error  correction  probabilities  uill  be  computed  and  correlated 
uith  the  3-d imensional  cell  matrix  link-reliability  score. 

Both  the  Data  Acquisition  task  and  Validation  task  have  been  designed 
usjng  the  sample  sizes  for  each  cell  of  the  3-dimensional  matrix  given 
in  "able  3.  Note  that  uith  the  exception  of  medium  and  high  in¬ 
formation  classification  rechecks*  cells  with  small  sample  sizes  have 
been  substantially  improved  from  those  in  Task  B.  These  rechecks 
were  not  improved  because  (1)  rechecks  have  been  found  to  be  inherently 
very  reliable  and  (2)  the  necessary  changes  to  the  tasks  to  increase 
these  sample  sizes  would  increase  task  length  andi  therefore!  increase 
training  time!  testing  time!  data  reduction  and  analysis  time! 
with  no  real  payoff. 
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TABLE  3 

DATA  ACQUISITION  TASK  SAMPLE  SIZES 
INFORMATION  HECAD  LINK  TYPE  TEICHNER  TRANSLATION  TYPE 


METRIC 

Conservation 

Creation 

Classif . 

Rechac  k 

6000 

Doffl  Path 

13200 

— 

— 

Low 

Mid-brarich 

5280 

— 

— 

Start/return 

720 

— 

— 

l-link 

Recheck 

. 

4560 

480 

Dorn  Path 

— 

2880 

6960 

Medium 

Mid-branch 

— 

1200 

1440 

Start/return 

— 

1920 

4800 

l-link 

1680 

1440 

Rechec  k 

5760 

Dom  Path 

— 

2880 

1440 

High 

Mid-branch 

— 

3360 

1440 

Start/return 

— 

2880 

1440 

l-link 

-  - 1 

2400 

1200 
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EXPERIMENTAL  ANALYSES 

For  all  37  trials<  of  both  tasks,  time  and  error  scores  mere  recorded. 
Time  scores  consisted  of  total  time  to  complete  each  trial  as  well 
as  individual  time  scores  for  each  matrix  switch  area  selected. 

Error  scores  consisted  of  a  listing  of  the  incorrect  matrix  switch 
areas  selected.  Total  number  of  errors  per  trial  as  well  as  the 
selected  incorrect  switch  area  were  also  recorded.  Also,  the  number 
of  error  feedback  and  error  intuitive  key  activations  were  recorded. 
From  these  totals,  mean  values  were  obtained  for  every  trial  <Ac- 
(^uisition  task.  nal2.  Validation  task.  n=8>.  Each  of  these  measures 
will  be  discussed  separately. 

Learnina/Performance  Curves 

As  can  be  seen  from  Figure  4.  time  and  error  scores  of  the  Data 
Acd,uisition  task  are  relatively  high  for  the  beginning  trials; 
however  both  of  these  measures  decrease  rapidly  then  maintain  a 
steady  level.  From  this  figure  it  was  determined  that  asymptotic 
performance  was  reached  by  trial  13  with  occasional  time  and  error 
fluctuations  due  to  breaks  and  lunch  periods. 

In  order  to  further  examine  the  time  performance  scores,  the  total 
time  per  trial  was  partitioned  into  two  component  parts;  namely,  the 
time  between  two  correct  responses  (C->C>  and  the  time  between  a 
correct  response  and  an  incorrect  response  (C-N).  Figure  5  illus¬ 
trates  this  relationship.  As  can  be  seen,  the  time  between  two 
correct  actions  was  very  consistent  over  all  trials  with  little  or 
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Figure  4.  Performance  time  and  errors  as  a  function  of  trials 
for  the  data  acquisition  task. 


Performance  time  for  correct  and  incorrect  responses  as  a  function  of 
trials  for  the  data  acquisition  task. 
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no  evident  time  fluctuations.  It  is  also  interesting  to  note  that 
there  is  little  evidence  of  learning  phenomena.  The  curve  for  the 
C-M  responses  reflects  the  amount  of  time  the  subject  took  when 
making  a  primary  error<  i. e. >  the  first  incorrect  response  in  any 
error  sequence.  This  curve  clearly  shouts  a  classic  learning  phase 
followed  by  asymptotic  performance.  One  would  expect  that  there 
would  be  a  greater  amount  of  time  taken  for  a  C-U  response  than  for 
a  C-C  response  due  to  the  subject  not  being  able  to  immediately  re¬ 
call  the  next  correct  action  to  be  made.  A  test  of  this  comparison 
for  asymptotic  trials  26-37  confirmed  the  expectation)  F( i< 22)»22.  83 
a.  <  .  01  <C-C  response  mean  =  1.65/  C-W  response  mean  =  2.48). 

The  validation  time  scares  were  summarized  in  the  same  manner  as 
the  acquisition  data.  Since  the  tasks  had  similar  values  for  the 
integrated  model  variables/  it  was  predicted  that  the  time  score 
relationships  would  be  similar.  Figure  6  depicts  the  mean  number 
of  errors  and  mean  time  as  a  function  of  trials.  As  can  be  seen/ 
there  is  a  decrease  in  time  and  error  performance  over  trials#  with 
asymptote  being  reached  by  approximately  trial  13.  In  order  to 
further  examine  the  time-error  relationships#  the  C-C  responses  and 
the  C-M  responses  were  isolated.  As  with  the  Acquisition  data#  it 
was  found  that  the  C-C  responses  (mean  »  1.77)  were  substantially 
faster  over  asymptotic  trials  than  were  the  C-U  responses (meana2.  92) 
F(li22)  «  157.80#  B,  <  .01.  This  relationship  is  illustrated  in 
Figure  7. 

An  additional  analysis  was  performed  for  trials  26-37  of  the  Acquisi 
tion  task  only.  This  consisted  of  comparing  the  response  time  for 
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Performance  time  and  errors  as  a  function  of  trials  for  the  validation  task. 


Performance  time  for  correct  and  incorrect  responses  as  a  function  of 
trials  for  the  validation  task. 
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the  first  to  second  incorrect  response  uith  the  response  times 
among  all  incorrect  strings  of  responses.  This  comparison  reached 
a  conventional  level  of  significance!  F  (l!22>  »  19.  s?/  g.  <  .  01  uith 
the  time  between  the  first  two  incorrect  responses  being  less  than  the 
time  between  subseq,uent  incorrect  responses.  A  reasonable  explana¬ 
tion  for  this  finding  concerns  the  procedure  for  signaling  subjects 
that  an  error  had  been  made.  This  signal  came  only  after  a  string 
of  two  errors  had  been  made.  It  is  suggested  that  on  many  occasions 
the  subjects  were  unaware  of  having  made  an  error  until  the  signal 
came  oni  and  hence  performed  the  second  error  in  the  string  as  rapidly 
as  if  no  error  had  been  made.  For  the  comparison  of  correct-correct 
responses  with  incorrect  #l-incorrect  #2  responses  there  was  no  differ 
ence  (F  <  l)i  with  the  correct-correct  responses  being  slightly 
faster.  However/  once  the  error  signal  came  on,  response  perfor¬ 
mance  was  slowed  by  the  cognitive  processes  of  attempting  error  correc 
tion. 


Error  Detection  and  Correction  Time  Scores 


As  described  in  the  Method  section/  two  matrix  switches  were  dedicated 
to  signaling  a  feedback  detection  and  an  intuitive  detection.  It 
was  initially  hypothesized  that  the  detection  of  a  primary  error 
detected  intuitively  would  take  less  time  than  an  error  detected 
through  feedback.  Figures  8  and  9  show  a  performance  tendency  that 
favors  this  hypothesis  during  preasymptotic  trials/  but  not  for  trials 
after  asymptote  was  reached.  There  were  no  significant  differences 
on  tests  run  for  asymptotic  trials  26-37/  so  statistical  tests  were 
run  using  all  trials.  For  the  Acq.uisition  task/  it  was  found  that 
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there  uas  a  significant  difference.  F{1<72)  *  6.  84»  a.  <  .03.  between 
the  incorrect  response  followed  bg  a  feedback  detection  and  an  in¬ 
correct  response  followed  by  an  intuitive  detection.  This  result 
was  in  the  predicted  direction  as  it  took  longer  (approximately  4 
sec)  to  signal  that  an  error  was  made  when  the  subject  had  to  rely 
on  feedback  information.  This  finding  was  substantiated  on  the 
Validation  task.  F(1.72)  »  5.52.  g.  ^  05.  with  feedback  detected 

errors  requiring  more  time  than  intuitively  detected  errors.  These 
findings  indicate  that  the  processes  involved  in  detecting  an  error 
by  means  of  feedback  are  complicated  enough  to  consume  a  substantial 
amount  of  time,  but  are  of  a  nature  that  is  susceptible  to  a  learning 
that  reduces  the  time  to  a  period  comparable  to  the  intuitive  process. 

For  the  correction  portion  of  an  error,  it  was  initially  hypothe¬ 
sized  that  it  would  take  a  greater  amount  of  time  to  correct  an 
error  that  was  proceeded  by  the  feedback  detection  key  than  the 
intuitive  detection  key.  There  were  no  significant  differences  on 
tests  run  over  asymptotic  trials  26-37.  While  the  results  for  over¬ 
all  trials  on  the  Acquisition  task  are  in  the  correct  direction, 
significance  was  not  obtained.  F<1.72)  =  3.80.  a.  >  .05.  For  the 
Validation  task,  however,  significance  was  reached.  F(l.72)  *  5.65. 
a.  <  .05.  with  feedback  detected  errors  requiring  substantially  more 
time  than  intuitively  detected  errors  prior  to  reaching  the  next 
correct  response.  Both  results  are  depicted  graphically  in  Figures 
10  and  11.  For  both  tasks  the  time  differences  are  largely  attribu¬ 
table  to  the  subjects'  preasymptotic  performance.  Once  asymptote 
was  reached,  the  differences  in  time  between  intuitive  and  feedback 
for  correction  of  an  error  wore  reduced.  Explanation  of  this  "cor- 
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Figure  10.  Time  to  correction  following  a  feedback  detection  and  an  intuitive 
detection  as  a  function  of  trials  for  the  Data  Acquisition  task. 


O  FEEDBACK  -  CORRECT  RESPONSE 


Time  to  correction  following  a  feedback  detection  and  an  intuitive 
detection  as  a  function  of  trials  for  the  validation  task. 
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raction"  finding  is  similar  to  the  explanation  advanced  for  the 
"detection”  finding<  in  that  both  represent  an  early  trial  incre¬ 
ment  of  processing  time>  associated  utith  detection  via  feedback/ 
that  dissipates  uiith  learning  trials.  Mhat  is  not  clear  from  the 
data  is  whether  the  subjects  became  more  efficient  in  their  capability 
to  scan  and  locate  display  items#  or  to  process  cognitive  material# 
or  both. 

Related  Findings 

According  to  Rabbitt  (1969)  the  response  time  to  the  first  event 
after  an  error  is  very  slow#  with  the  responses  to  the  next  few 
events  also  being  slower  than  average.  Furthermore#  Rabbitt  <1966a# 
1966b.  1968)  found  that  errors  were  much  faster  than  correct  re¬ 

sponses.  Subjects  make  runs  of  increasingly  fast  responses#  which 
terminate  in  an  even  faster  error.  Evidence  for  none  of  these  find¬ 
ings  was  found  in  the  present  study  which  in  fact  showed  opposite 
effects.  Rabbitt  used  a  choice  reaction  time  task#  while  the  pre¬ 
sent  study  had  a  stronger  cognitive  task  element  with  a  heavy  de¬ 
pendence  on  long  term  memory.  It  is  apparently  this  task  type  dif¬ 
ference  that  accounts  for  the  differences  in  times  associated  with 
error  events. 

Node  vs  Link  Errors 

Findings  from  previous  studies  (Goldbeck  Charlet#  1975)  have  shown 
that  task  time  and  reliability  are  determined  by  task  structure.  In 
that  study  it  was  shown  that  task  structure  characteristics  can  be 
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axprassed  in  tarrns  of  parameters  describing  how  a  system  task  at 
a  control-display  panel  is  performed  differently  on  different 
occasions.  In  order  to  develop  possible  prediction  algorithms  for 
operator  performance>  as  well  as  error  detection  and  correction<  the 
task  must  be  represented  as  a  network  of  control  and  display  links. 
Before  presenting  the  actual  results<  a  brief  summary  will  be  de¬ 
voted  to  the  models  for  operator  performance  and  how  the  actual  values 
for  the  control  and  display  links  are  derived. 

Performance  models.  Three  different  models  were  combined  in  an 
attempt  to  predict  operator  task  times  and  error  rates.  The  first 
model  was  developed  by  AMRL  in  cooperation  with  Ford  Aerospace*  and 
is  referred  to  as  HECAD  <Human  Engineering  Computer  Aided  Design). 

This  model  assigns  values  to  either  control  or  display  links.  From 
each  control  or  display  (node)  there  are  one  or  more  links  to  other 
nodes.  For  each  link  there  are  different  associated  probabilities 
that  an  action  will  occur.  The  path  of  highest  link  probablity 
through  the  task  network  is  the  dominant  path.  When  performance 
of  the  task  departs  from  the  dominant  path  of  high  probability  links* 
these  departures  arc  referred  to  as  branch  paths.  In  addition  to 
the  dominant  path  links*  there  are:  1)  the  first  and  last  links 
of  a  branch*  or  start-returns*  2)  the  middle  links  of  a  branch* 
each  called  a  mid-branch*  3)  1-link  branches*  i. e*  a  non-sequen- 
tial  jump  from  one  dominant  path  node  to  another*  and  4)  rechecks. 

The  recheck  link  category  is  not  mutually  exclusive  with  respect  to 
the  other  four  types  but  this  designation  supercedes  all  other  cate¬ 
gories.  All  link  errors  were  assigned  to  the  appropriate  link  type. 
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Tha  sacor>d  modal  for  parformanca  loas  daveloped  bg  Narran  Taichnar. 

The  three  corresponding  features  for  this  modal  are:  1)  conserva¬ 
tion.  signifies  a  one  to  one  relationship.  2)  classification,  a  many 
to  feu  relationship,  and  3)  creation,  a  feui  to  many  relationship. 

All  link  errors  mere  assigned  to  the  appropriate  Teichner  transla¬ 
tion  type. 


The  third  model  uias  an  Information  hetric  value  based  upon  the  for¬ 
mula  _  r  p  loop  •  The  resultant  values  mere  divided  into  three  link 
"  i  “2  i 

types:  1)  high  information  content.  2)  medium  information  content, 

and  3)  lou  information  content.  These  three  link  characteristics 
uere  then  combined  into  a  3-dimensional  matrix  uiith  reliabilities 
computed  for  each  cell. 


Link-node  assignments.  All  asymptotic  trial  errors  were  divided 
into  two  types,  namely,  link  errors  and  node  errors.  Node  errors 
result  when  errors  of  manipulation  occur  at  the  correct  or  adjacent 
control  or  display,  i.  e.  .  the  operator  was  at  the  correct  point 
in  the  task  conceptually,  but  did  not  sucessfully  perform  the  correct 
action.  Link  errors  were  the  result  of  an  incorrect  cognitive 
transfer  from  the  last  node  he  sucessfully  performed  to  the  next 
node. 


Once  an  error  was  assigned  to  a  link  or  node  it  was  then  assigned  to 
a  control  or  display.  Display  errors  were  categorized  by  the  follow¬ 
ing  parameters:  1)  control  is  correct  for  different  status  of  dis¬ 
play.  2)  control  is  not  correct  for  any  state  of  proceeding  display. 
3)  control  is  correct  for  an  adjacent  display  of  the  same  type  and 
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statusi  and  4)  tima  limit:  no  control  is  salactad.  Sifflilarly>  errors 
mare  catagorizad  as  control  errors  whan  any  one  of  the  following  criteria 
was  met:  1)  similar  control  namest  but  an  incorrect  or  adjacent  control 
is  touched!  2)  control  preceded  by  another  control  with  no  intervening 
display#  3)  similar  control  name  and  in  vicinity  of  correct  control# 
and  4)  time  limit:  a  series  of  incorrect  controls  are  selected. 


Overall  reliabilitu  results.  For  both  tasks  all  link  errors  were 
categorized  into  a  three  dimensional  matrix  using  the  HECAD#  Teichner# 
and  Information  Metric  values.  From  this  matrix  an  overall  link 
error  rate  table  was  derived.  Proportions  were  calculated#  based 
on  the  total  number  of  link  errors  per  cell#  corrected  for  the  cell 
frequency#  number  of  subjects#  and  trials.  Lastly#  a  reliability 
measure  was  calculated  for  each  cell. 

As  expected  from  Task  B  results  of  the  previous  study#  HECAD  link  type 
summaries  of  asymptotic  trials  from  the  Data  Acquisition  task  in 
Table  4  shows  the  same  order  from  most  to  least  reliable  links:  re- 
check#  dominant  path#  mid-branch#  start/return#  and  1-link.  The 
rationale  for  these  findings  with  the  empirically  derived  categories 
includes  the  fact  that  rechecks  should  have  been  easy  because  they 
were  strongly  cued  by  an  identical  response.  Dominant  path  links 
should  have  been  easy  due  to  their  high  probability  of  occurrence 
and  the  fact  that  they  were  the  nominal  response  in  the  path  of  task 
resonses.  Mid-branches  were  moderately  well  cued  by  preceding  branch 
links#  but  start/returns  represented  a  shift  to  or  from  a  branch. 

The  1-link  was  a  low  probability  event  that  revised  normal  dominant 
path  relationships.  A  parallel  rationale  for  the  results  concern 
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Table  4 


Overall  Link  Reliability  Rates:  Acq.uisition  Task 


INFO  HECAO  LINK 

METRIC  TYPE 

TEICHNER  TRANSLATION 
Conservation  Creation 

TYPE 

Class 

SUMMARY 

Rechec  k 

— - ] 

1.  0000 

— 

— 

Dorn  Path 

.  9915 

— 

— 

Low  Mid-branch 

.  9965 

— 

— 

.  9946 

Start/return 

.  9907 

— 

— 

1-1 ink 

•••• 

Recheck 

.  9901 

.  9545 

Doffl  Path 

— 

.  9873 

.  9890 

Medium  Mid-branch 

— 

.  9911 

.  9889 

.  9845 

Start/return 

— 

.  9705 

.  9740 

1-1 ink 

" 

.  9708 

.  9479 

Recheck 

1.  0000 

— 

Ooffl  Path 

— 

.  9878 

.  9919 

High  Mid-branch 

— 

.  9787 

.  9444 

.  9746 

Start/return 

— 

.  9311 

.  9734 

1-link 

—— 

.  9576 

.  9639 

SUMMARY 

.  9946 

.  9811 

.  9783 

HECAD  LINK  TYPE  SUMMARIES 


Rechec  k 

.  9960 

Dorn  Path 

.  9901 

Mid-branch 

.  9951 

Start/return 

.  9667 

1-link 

.  9612 
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the  general  correspondence  of  response  reliability  with  the  relative 
amount  of  practice  within  the  task.  The  Validation  task  results  for 
HECAD  link  types  shown  in  Table  5  have  the  predicted  order  of  relia¬ 
bility  with  the  exception  of  a  reversal  between  start/return  and  1- 
1  ink. 

For  both  tasksi  the  low  information  metric  category  resulted  in  the 
highest  reliability  value  and  the  high  information  metric  category 
resulted  in  the  lowest  reliability  value.  This  finding  is  in  keeping 
with  the  expectation  that  higher  uncertainty  will  be  associated  with 
mors  errors. 

For  the  Teichner  classif ications  it  was  found  that  the  conservation^ 
that  isi  a  node  with  one  state  and  one  output*  yielded  a  higher  re¬ 
liability  than  the  creation  or  c lassif ication  for  both  tasks.  A 
surprising  finding  is  that  the  Creation  translation  was  more  reliable 
than  classification  translation  for  both  tasks*  as  well  as  for  the 
previous  Task  B.  It  would  have  been  more  plausible  if  the  creation 
had  been  least  reliable  because  multiple  outputs  are  possible  for 
it*  depending  on  the  status  of  a  previous  node.  For  a  classification 
there  is  only  one  output  possible  for  a  multi-state  node. 

Despite  the  similarity  between  the  Data  Aci^uisition  and  Validation 
tasks  in  regard  to  their  having  the  same  dominant  path  and  branch 
sequences*  their  overall  reliability  differed  substantially.  The 
Data  Acquisition  task  had  a  reliability  of  .9850  and  the  Validation 
task  had  a  .9775  reliability.  It  is  speculated  that  the  most  likely 
area  for  finding  parameters  that  would  account  for  a  major  part  of 
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Table  5 


Overall  Link  Reliability  Rates:  Validation  Task 


INFO  HECAD  LINK 

TEICHNER  TRANSLATION  TYPE 

SUMMARY 

METRIC  TYPE 

Conservation 

Creation  Class 

Rechec  k 

.  9987 

Dom  Path 

.  9962 

— 

— 

Loui  Mid-branch 

.  9915 

— 

— 

.  9953 

Start/return 

.  9792 

— 

— 

1-link 

Recheck 

— 

1 .  0000 

1 .  0000 

Dom  Path 

— 

.  9818 

.  9810 

Medium  Mid-branch 

— 

.  9524 

.  9927 

.  9720 

Start/return 

— 

.  9661 

.  9292 

1-link 

— — 

.  9485 

.  9740 

Rechec  k 

1 .  0000 

Dom  Path 

— 

.  9627 

.  9983 

High  Mid-branch 

— 

.  9754 

.  9410 

.  9645 

Start/return 

— 

.  9167 

.  9549 

1-link 

—— 

.  9135 

.  9833 

SUMMARY 

.  9953 

.  9695 

.  9833 

HECAD  LINK  TYPE  SUMMARIES 


Rechec  k 

.  9996 

Dom  Path 

.  9874 

Mid-branch 

.  9781 

Start/return 

.  9384 

1-1  ink 

.  9422 
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this  diffarence  is  the  area  of  task  scenario>  or  story  line.  Such 
characteristics  as  simplicity  and  redundancy  could  be  scaled. 

Detection  reliabilities.  Tables  6  and  7  show  the  detection 
proportions  for  intuitively  detecting  an  error.  These  proportions 
were  calculated  by  taking  the  total  number  of  intuitive  detections 
and  dividing  this  total  by  the  number  of  link  errors. 

For  both  the  Ac(iuisition  and  Validation  tasks*  the  high  categorization 
under  the  Information  Metric  resulted  in  the  highest  reliability. 

For  the  medium  and  low  information  categories  there  was  no  consis¬ 
tency  between  tasks*  with  the  low  information  metric  having  a  higher 
reliability  than  the  medium  for  the  Acq.uisition  task  and  the  con¬ 
verse  situation  for  the  Validation  task. 

The  Teichner  classifications  did  not  reveal  any  predictive  or  con¬ 
sistent  measures  between  the  two  tasks.  The  Validation  task  shows 
that  a  link  error  that  occurred  on  a  node  with  multiple  outputs  was 
more  likely  to  be  detected  than  a  node  error  where  there  was  only  one 
output.  This  is  also  upheld  for  the  Acquisition  task  as  the  crea¬ 
tion  category  shows  a  higher  detection  rate  than  the  classification 
category.  However*  unlike  the  Validation  task*  the  conservation 
category  yielded  an  even  higher  error  reliability  figure  than  both 
the  creation  and  classification  categories*  thus  making  interpretation 
difficult.  This  increase  in  intuitive  detection  of  an  error  could  be 
attributable  to  the  fact  that  a  subject  had  several  sources  of  infor¬ 
mation  available  to  him  in  order  to  cue  him  that  something  had  gone 
awry  in  the  operating  system.  Similarly*  if  the  subject  had  only 
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Table  6 


Proportions  for  the  Intuitive  Detection 
of  Primary  Error:  Acquisition  Task 


INFO 

HECAD  LINK 

TEICHNER  TRANSLATION 

TYPE 

SUMMARY 

METRIC 

TYPE 

Conservation  Creation  Class 

Rechec  k 

.  0000 

Oom  Path 

.  26S7 

— 

— 

Loui 

Mid-branch 

.  3636 

— 

— 

.  2821 

Star t/return 

.  0000 

— 

— 

1-link 

" 

“ 

Rechec  k 

... 

.  0000 

.  4616 

Dorn  Path 

— 

.  5909 

.  6522 

Med ium 

Mid-branch 

— 

.  3333 

.  6250 

.  4784 

Start/return 

— 

.  6364 

.  4533 

1-link 

.  6667 

.  3333 

Rechec  k 

.  0000 

Oom  Path 

— 

.  4762 

.  2857 

High 

Mid-branch 

— 

.  4419 

.  1250 

.  2615 

Start/return 

— 

.  2017 

.  2074 

1-link 

— — 

,  3115 

.  2308 

SUMMARY 

.  2821 

.  3263 

.  3801 

HECAD  LINK  TYPE  SUMMARIES 


Recheck 

.  1500 

Dom-Path 

.  4479 

Mid-branch 

.  3307 

Start/return 

.  3017 

l-link 

.  3577 
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Table  7 


Proportions  for  the  Intuitive  Detection 
of  a  Primary  Error;  Validation  Task 


INFO  HECAD  LINK 

TEICHNER  TRANSLATION 

TYPE 

SUMMARY 

METRIC 

Conservation 

Creation 

Class 

Recheck 

.  6667 

— 

Dorn  Path 

.  4500 

— 

— 

Lota  Mid-branch 

.  5000 

— 

— 

.  4861 

Star t/r eturn 

.  3333 

— 

— 

1-link 

—  — 

**•“ 

Recheck 

.  0000 

.  0000 

Dorn  Path 

— 

.  5238 

.  1698 

Medium  Mid-branch 

— 

.  0000 

.  0000 

.  2810 

Star t/return 

— 

.  5769 

.  3088 

1-link 

.  2692 

.  4000 

Recheck 

— • 

.  0000 

— 

Dorn  Path 

— 

.  1628 

.  0000 

High  Mid-branch 

— 

.  4848 

.  1471 

.  1883 

Start/return 

— 

.  1354 

.  3077 

1-link 

— 

.  1084 

.  3750 

SUMMARY 

.  4681 

.  2167 

.  2556 

- -  ■  .e 

HECAD  LINK  TYPE  SUMMARIES 


Recheck 

.  6667 

Dorn  Path 

.  2609 

Mid-branch 

.  2419 

Start/return 

.  2759 

1-link 

.  1721 
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one  clue  in  the  operating  system  that  an  error  had  occurred<  this 
clue  may  be  overlooked!  thereby  louering  the  detection  reliability. 


As  uiith  the  Tiechner  classifications!  HECAD  reliability  figures  mere 
not  consistent  between  tasks.  Due  to  this  inconsistency  it  is  diffi¬ 
cult  to  draw  any  substantial  conclusions  regarding  this  variable 
(see  Tables  6  and  7). 


In  addition  to  just  detecting  an  error  there  is  also  detection  im¬ 
mediately  followed  by  a  correct  response.  Tables  8  and  9  summarize 
the  proportions  for  the  intuitive  detection  of  a  primary  error 
resulting  in  a  correct  response.  This  aspect  will  be  briefly  dis¬ 
cussed  by  its  impact  on  Teichner!  HECAD!  and  the  Information  Metric 
categories. 

Results  show  that  under  the  Teichner  categories  the  classification 
(many  to  few)  grouping  yielded  the  largest  proportion  of  detections 
with  correction  for  both  tasks.  While  this  finding  is  consistent 
between  tasks!  the  conservation  and  classification  categories  were 
in  opposite  directions.  Due  to  this  inconsistency!  interpretation 
is  difficult  with  the  amount  of  information  we  presently  have. 

For  the  HECAD  categories  there  was  no  consistency  between  the  two 
tasks.  Due  to  this  inconsistency!  no  conclusions  or  speculations 
will  be  made  regarding  this  variable.  The  obtained  results  for 
the  Information  Metric  were  similar!  but  only  for  the  medium  in¬ 
formation  category.  For  both  tasks!  the  medium  category  resulted 
in  a  high  proportion  of  errors  being  detected  and  corrected.  For 


49 


Ford  Aorospaco  A 
Communleatlona  Corporation 


Tabla  8 


) 

I 


Proportions  for  Intuitive  Detection  of  a  Primary 
Error  Resulting  in  a  Correct  Response:  Acq^uisition  Task 


INFO  HECAD  LINK  TYPE 

TEICHNER  TRANSLATION  TYPE 

SUMMARY 

METRIC 

Conservation  Creation  Class 

Recheck 

.  0000 

Dorn  Path 

.  7778 

— 

— 

Low  Mid-branch 

1.  0000 

— 

— 

.  6923 

Start/return 

.  0000 

— 

— 

1-1  ink 

—— 

— - 

— 

Recheck 

.  0000 

.  0000 

Dorn  Path 

— 

1.  0000 

1 . 0000 

Medium  Mid-branch 

— 

1 .  0000 

1 .  0000 

.  9344 

Start/return 

— 

1 . 0000 

1 .  0000 

1-link 

.  7143 

.  8000 

Rechec  k 

... 

.  0000 

h - 

Dorn  Path 

— 

1 .  0000 

.  5000 

High  Mid-branch 

— 

.  5789 

.  6667 

.  7692 

Start/return 

— 

.  7500 

1 .  0000 

1-link 

— 

.  7895 

1 .  0000 

SUMMARY 

.  6923 

.  7982 

.  9327 

HECAD  LINK  TYPE  SUMMARY 


Recheck  1.0000 
Dorn  Path  . 9315 
Mid-branch  .  7619 
Start/return  .  8243 
1-link  .7955 
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Table  9 


Proportions  for  Intuitive  Detection  of  a  Primary 
Error  Resulting  in  a  Correct  Response;  Validation  Task 


INFO  HECAD  LINK 

TEICHNER  TRANSLATION  TYPE 

SUMMARY 

METRIC 

Conservation 

Creation  Class 

Rech  ec  k 

.  5000 

— 

— 

Dorn  Path 

.  8889 

— 

— 

Low  Mid-branch 

.  8889 

— 

— 

.  8636 

Star t/r eturn 

1.  0000 

— 

— 

1-link 

Rechec  k 

— 

.  0000 

Km 

Dorn  Path 

— 

.  9091 

Medium  Mid-branch 

— 

.  0000 

.  9651 

Star t/return 

— 

.  9333 

1 .  0000 

1-link 

— — 

1.  0000 

1.  0000 

Recheck 

Dorn  Path 

— 

High  Mid-branch 

— 

.  7705 

Star t/return 

— 

1 .  0000 

1-link 

.  6667 

1.  0000 

SUMMARY 

.  8636 

.  8333 

.  9420 

HECAD  LINK  TYPE  SUMMARY 


Rechec  k 

.  5000 

Dorn  Path 

.  8889 

Mid-branch 

.  7000 

Start/return 

.  9625 

1-link 

.  8571 

Communications  Corporation 

the  low  and  high  information  categories  there  was  no  consistency 
between  tasks;  with  the  high  information  metric  having  a  higher  de¬ 
tection  and  correction  proportion  than  the  low  information  metric 
for  the  Acq.uisition  task  and  the  converse  situation  for  the  Valida¬ 
tion  task. 

Feedback  detected  errors.  As  mentioned  previously^  a  feedback  detected 
error  refers  to  the  fact  that  the  subject  re^iuired  some  outside 
or  additional  information  to  inform  him  that  an  error  had  been 
made.  Tables  10  and  11  summarize  these  findings.  For  the  Information 
Metric  variable  there  was  no  consistency  between  tasks  for  the  pro¬ 
portions  or  reliabilities  of  detected  errors.  One  point  that  should 
be  made  however«  is  that  there  are  relatively  few  errors  that  come 
under  the  feedback  detected  error  category  (i.e. <  17  out  of  677); 
therefore  it  is  extremely  difficult  to  draw  any  definitive  conclu¬ 
sions.  For  the  Teichner  categoriesr  both  tasks  resulted  in  the  M2 
c lassif ication  (many  to  few)  yielding  the  highest  reliability.  An 
examination  of  Tables  10  and  11  show  that  caution  should  be  taken  with 
this  interpretation  as  it  is  based  on  only  six  observations#  thus 
making  interpretations  of  the  data  difficult  to  make.  The  HECAD 
categories  yielded  little  insight  into  the  feedback  detection  q.ues- 
tion.  Again#  this  inconsistency  between  tasks  is  basically  attri¬ 
butable  to  the  small  number  of  errors  detected  through  feedback. 

In  addition  to  detection  of  an  error  with  the  feedback  key#  it  was 
decided  to  examine  the  correction  rate  immediately  following  the 
feedback  error  key.  The  proportion  of  errors  that  were  initially 
activation  of  the  detected  and  feedback  corrected  was  high  for  both 
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Table  10 


Proportions  for  the  Feedback  Detection 
of  a  Primary  Error;  Acq.uisition  Task 


INFO  HECAD  LINK  TYPE 

METRIC 

TEICHNER  TRANSLATION 
Conservation  Creation 

— 

TYPE 

Class 

SUMMARY 

Rechec  k 

.  0068 

— — 

— 

Dom  Path 

.  0485 

— 

— 

Low  Mid-branch 

.  0000 

— 

— 

.  0175 

Start/return 

.  0000 

— 

— 

1-link 

“ 

“ 

Recheck 

— 

.  0631 

.  0000 

Dom  Path 

— 

.  0000 

.  0000 

Medium  Mid-branch 

— 

.  0000 

.  0000 

.  0497 

Start/return 

— 

.  5000 

.  0000 

1-link 

“ 

.  0000 

.  0000 

Recheck 

.  0435 

Dom  Path 

— 

.  0000 

.  0000 

High  Mid-branch 

— 

.  0000 

.  0000 

.  0192 

Start/return 

— 

.  0000 

.  0000 

1-link 

“ 

.  0000 

.  0000 

SUMMARY 

.  0175 

.  0495 

.  0000 

HECAD  LINK  TYPE  SUMMARY 


Recheck 

.  0335 

Dom  Path 

.  0485 

Mid-branch 

.  0000 

Start/return 

.  0108 

1-link 

.  0000 
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Table  11 


Proportions  for  the  Feedback  Detection 
of  a  Primary  Error:  Validation  Task 


INFO  HECAD  LINK  TYPE 

TEICHNER  TRANSLATION 

TYPE 

SUMMARY 

METRIC 

Conservation 

Creation 

Class 

Recheck 

.  0000 

— 

— 

Dorn  Path 

.  0500 

— 

— 

Low  Mid-branch 

.  0000 

— 

— 

.  0362 

Star t/return 

.  1667 

— 

1-link 

— 

— 

—  — 

Recheck 

.  0000 

.  0000 

Dorn  Path 

— 

.  0476 

.  0377 

Medium  Mid-branch 

— 

.  0313 

.  0000 

.  0327 

Star t/return 

— 

.  0000 

.  0221 

1-link 

— 

.1154 

.  0000 

. 

Recheck 

WK  •  • 

HBIHI 

Dorn  Path 

— 

BKi  1  c 

High  Mid-branch 

— 

c  • 

.  0154 

Star t/return 

— 

.  0208 

1-link 

.  0000 

.  0000 

SUMMARY 

.  0362 

.  0250 

.  0222 

HECAD  LINK  TYPE  SUMMARY 


Rechec  k 

.  0000 

Dorn  Path 

.  0362 

Mid-branch 

.  0242 

Start/return 

.  0207 

1-link 

.  0246 
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As  defined  previously!  node  errors  are  those  errors  uthich  can  be 
attributed  to  a  mismanipulation  of  the  matrix  area.  For  the  Acq.uisi- 
tion  taski  post-asymptotic  trials  onlyi  reliability  figures  mere  ob¬ 
tained  for  the  three  variables;  namely<  the  Information  MetriCi 
Teichner  classifications#  and  HECAD  categories.  It  uias  originally 
hypothesized  that  the  node  error  rate  mould  not  lend  itself  to  be 
a  viable  predictor  variable  for  operator  performance*  but  would 
prove  to  be  an  overall  baseline  error  rate.  This  baseline  error 
rate  would  be  representative  of  those  errors  that  would  occur 
during  operating  periods  due  to  mismanipulations  of  the  equipment* 
lapses  of  attention  (i.e. *  time  out  errors)*  or  inadvertently  acti¬ 
vating  an  adjacent  switch  area. 

An  examination  of  Table  12  shows  that  for  the  HECAD  categories 
there  was  no  significant  difference  between  the  types  of  links. 
Therefore*  this  baseline  error  rate  and  the  reliability  estimates 
are  monotonic  with  respect  to  the  varying  levels  and  difficulty 
of  the  HECAD  categories.  This  finding  also  extends  to  the  Infor¬ 
mation  Metric  and  the  Teichner  categories.  An  inspection  of  the 
overall  reliabilities  shows  that  the  occurrence  of  a  node  error  was 
likely  but  at  a  low  level.  Additionally*  they  were  normally  dis¬ 
tributed  throughout  the  task*  i. e. *  there  was  no  particular  bias 
by  category  (Acquisition  task  reliability  *  .9978*  Validation  task 
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Table  12 


Node  Reliability  Values  for  the 
Data  Acquisition  Task 


INFO 

METRIC 


HECAD  LINK 


TEICHNER  TRANSLATION  TYPE 
Conservation  Creation  Class 


SUMMARY 


Recheck 
Dodi  Path 

Louf  Mid-branch 

Star t/return 
1-link 


1.  0000 
.  9995 
.  9981 
.  9970 


9989 


Med ium 


Recheck 
Dorn  Path 
Mid-branch 
Start/return 
1-link 


1.  0000 
.  9931 
.  9875 
.  9939 
.  9875 


1.  0000 
.  9943 
1.  0000 
.  9983 
1.  0000 


9965 


Recheck 
Dorn  Path 

Hiyh  Mid-branch 

Start/return 

1-link 


1 .  0000 
.  9948 
.  9985 
.  9959 
.  9948 


.  9977 
.  9977 
1 .  0000 
.  9977 


9982 


SUMMARY 


9989 


.  9973 


.  9972 


HECAD  LINK  TYPE  SUMMARY 


Recheck  1.0000 
Dom  Path  .  9969 
Mid-branch  .9981 
Start/return  .  9970 
1-link  .9968 
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It  was  originally  hypothesized  that  detection  and  correction  pro¬ 
babilities  uere  a  function  of  HECAD  link  types.  Teichner  transla¬ 
tion  types,  and  the  amount  of  information  processed.  Like  perfor¬ 
mance  reliabilities,  error  detection  and  correction  data  could  be 
examined  in  a  similar  manner.  In  the  present  study  houever.  the  error 
sample  size  is  a  subset  of  the  total  for  a  given  cell  in  the  three 
dimensional  matrix.  The  resultant  error  detection  and  correction 
sample  size  uas  insufficient  to  determine  necessary  prediction  al¬ 
gorithms.  Therefore,  the  techniiLue  of  comparing  the  predicted  with 
the  actual  value  for  each  cell  was  not  successfully  used  to  examine 
detection  and  correction  probabilities.  Instead,  the  error  detection 
and  correction  probabilities  were  computed  and  plotted.  Using  scatter 
plots,  correlation  coefficients  were  calculated  using  the  cell  perfor¬ 
mance  reliability  and  the  probability  of  detection  of  an  error  on  the 
Data  Acquisition  task,  as  well  as  the  Validation  task.  This  design 
yielded  six  correlations  per  task  based  on  the  Information  Metric  cate¬ 
gories. 

For  the  Data  Acquisition  task,  the  low  information  category  yielded 
the  largest  correlation  coefficient  <r«-4-79).  Before  drawing  any 
parallels  regarding  the  overall  error  and  the  detection  probabilities, 
one  should  look  at  the  other  correlation  coefficient  values  for  the 


remaining  information  categories  to  see  if  any  patterns  are  evident. 

For  the  medium  information  category  a  correlation  of  >.  08  was  obtained; 
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for  tha  high  information  category  the  correlation  value  was  67. 
Corresponding  correlations  were  run  for  the  correction  data<  but  they 
were  negative. 

Clearly  there  is  no  simple  interpretation  of  the  data.  One  possible 
explanation  for  the  low  information  category  yielding  a  high  correla¬ 
tion  is  the  assumption  that  it  was  very  easy  for  a  subject  to  detect 
that  an  error  had  occurred.  Thereforei  the  higher  the  reliability 
ratei  the  greater  the  chance  for  detection*  and  since  the  errors 
are  readily  apparent  to  the  operator  there  would  be  a  strong  rela¬ 
tionship  for  errors  being  detected.  This  explanation*  however*  does 
not  fully  explain  the  obtained  results*  therefore*  an  examination 
of  the  validation  correlation  coefficients  was  necessary.  The  low 
information  category  on  this  task  yielded  a  high  correlation  (■»■.  84)* 
while  the  correlations  for  the  medium  and  high  category  were  ■•-.25 
and  -•■.26  respectively.  The  additional  correlations  when  considered 
separately  did  not  lend  further  insight*  so  an  overall  correlation 
coefficient  was  calculated  for  each  task.  For  the  Acy.usition  task 
this  yielded  a  value  of  -•■.51*  for  the  Validation  task  a  value  of 
-•■.43  was  obtained.  Uhile  this  additional  information  does  not  pro¬ 
vide  a  solution*  it  does  show  that  the  degree  of  relationship  be¬ 
tween  the  two  variables  is  similar*  and  substantial  for  both  tasks. 

The  following  simple  algorithm  is  a  convenient  means  for  showing 
the  effects  of  detection  and  correction  on  task  reliability; 

R  «  R  ^  C(l-R, )  (D, )  (C,)! 

A  a  a  a  a 
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Mhcre  “  operator  reliabilitg  for  task  action  A.  i. e. .  the  pro¬ 
bability  that  task  action  A  mill  be  correctly  performed 
(including  error  detection  and  correction). 

R^  ™  The  probability  that  task  action  A  uill  be  performed 
uiithout  error. 

Dg  =  The  probability  that  the  operator  detects  an  error  in 
the  performance  of  task  action  A. 

s  The  probability  that  a  detected  error  is  successfully 
corrected  by  the  operator. 

While  in  the  present  study  there  was  not  found  any  prediction  func¬ 
tions  for  the  values  of  D  or  Ci  some  information  can  be  gained  by 
presenting  the  obtained  values  for  D  and  C  from  the  Data  Act^uisition 
and  Validation  tasksi  and  showing  how  they  affected  the  reliability 
value  of  the  algorithm. 

Substituting  in  the  algorithm  for  operator  reliability  from  the  Data 
Acq.uisition  task  gives  the  following: 

.  98S0  +  C(.  0150)  (.  6500)  (.  9231):  »  .9940 

The  .6500  value  for  detection  which  was  measured  before  the  error 
light  came  on>  can  be  partitioned  into  the  following  component  values 

.3456  from  cases  when  the  subject  signaled  the  detection  to 
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b«  intuitiva< 

.0265  from  cases  uhen  the  subject  signaled  the  detection  to  be 
from  feedbacki 


.2779  from  cases  when  the  subject  failed  to  signal  and  went 
directly  to  the  correction. 

For  cases  when  an  intuitive  detection  was  signaled^  the  obtained  pro¬ 
portion  of  correction  was  .  8596.  For  cases  when  a  feedback  detection 
was  signaled!  the  obtained  proportion  of  correction  was  .9444. 

Substituting  in  the  algorithm  for  operator  reliability  from  the  Vali¬ 
dation  task  gives  the  following: 

.9775  +  C  (.  0225)  (.  6544)  (.  9503)  «  .9915 

The  .6544  value  for  detection!  which  is  close  to  the  detection  value 
of  .  6500  for  the  Data  Acq.uisition  task!  can  be  partitioned  into  the 
following  values: 


.2496  from  cases  when  the  subject  signaled  the  detection  to 
be  intuitive! 

.0251  from  cases  when  the  subject  signaled  the  detection  to 
be  from  feedback! 

.3796  from  cases  when  the  subject  failed  to  signal  and  went 
directly  to  correction. 


For  all  cases  when  a  intuitive  detection  was  signaled!  the  obtained 
proportion  of  correction  was  .8824.  Values  for  the  Validation  task 
closely  followed  the  predictions  from  the  Data  Acq,uisition  task. 
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The  general  conclusion  from  results  of  both  tasks  is  that  task  re- 
liabiltv)  can  be  substantially  reduced  by  operator  detection  and  cor¬ 
rection.  For  tasks  similar  to  those  used  in  the  present  studyi  the 
detection  and  correction  values  from  this  study  could  be  used  if 
no  more  appropriate  data  are  available. 


QUEST IONA I RE  COMMENTS 

Immediately  follouing  the  last  data  collection  trials  all  subjects 
mere  asked  to  fill  out  a  questionnaire  giving  their  comments  and 
criticisms  regarding  the  present  study.  The  questions  asked  of  the 
subjects  centered  on  several  areas;  namelyi  the  training  and  data 
collection  trials*  use  of  the  error  keys*  comments  regarding  the 
hardware  equipment*  and  environmental  conditions.  Mhile  the  comments 
given  by  all  subjects  uere  noted  by  the  experimenter*  no  formal 
quantification  of  the  comments  was  done. 

There  was  total  agreement  by  all  subjects  that  the  training  and  data 
collection  trials  were  well  designed  and  facilitated  their  learning 
of  the  task.  Additionally*  ample  time  was  provided  for  breaks  and 
lunch  periods  thus  eliminating  boredom  or  fatigue  factors. 

The  next  set  of  questions  dealt  with  the  use  of  the  error" Teys*- “ 
spec  if  the  subjects  if  they  were  confused  about  which 

key  to  use  or  if  they  had  ever  used  the  wrong  error  key  accidentally. 
Seventy  percent  of  the  subjects*  on  both  tasks*  commented  that 
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there  uas  no  confusion  on  their  part  uith  uhich  error  key  to  use. 
Twenty  percent  of  the  subjects  said  that  initially  there  uas  some 
confusion  deciding  on  which  error  key  to  use  or  that  occasionally 
they  used  the  wrong  error  key.  All  subjects  in  this  category  fur¬ 
ther  stated  that  this  problem  was  eliminated  by  becoming  more  familiar 
with  the  task.  Two  subjects  ilOV.)  however>  stated  that  they  exper¬ 
ienced  confusion  discriminating  between  the  two  error  keys  through¬ 
out  the  entire  task  but  on  an  intermittent  basis.  Fallowing  this 
question!  subjects  were  asked  if  they  experienced  any  additional 
problems  associated  with  the  error  keys.  Fifty  percent  of  the 
subjects  commented  that  on  occasion  they  forgot  to  use  either 
error  key  when  an  error  was  made.  Similarly/  15  percent  said  that 
they  had  accidentally  hit  an  error  key  when  no  error  had  been  made. 

The  next  set  of  questions  dealt  with  the  criteria  the  subjects  had 
set  for  their  performance.  Forty  percent  of  the  subjects  were  trying 
to  reduce  their  error  rate  as  well  as  increase  their  speed.  The 
remainder  of  the  subjects  were  concentrating  on  one  particular  aspect/ 
namely/  reducing  the  error  rate  (35%)  or  decreasing  their  time  scores 
(25^).  Lastly/  all  subjects  were  asked  if  they  felt  that  they  could 
improve  their  performance  scores;  ninety  percent  of  the  subjects 
answered  affirmatively. 

In  addition  to  the  specific  questions  mentioned  above/  subjects  were 
given  the  opportunity  to  comment  on  other  aspects  of  the  experiment. 
The  majority  of  the  subjects  gave  favorable  comments  regarding  the 
touch  entry  CRT/  such  as  extremely  responsive  and  easier  to  use 
than  a  conventional  keyboard.  Several  suggestions/  such  as  adding 
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SECTION  4 

RESEARCH  IMPLICATIONS 

The  three-dimensional  model  of  HECAD  Categories/  Teichner's  Classi¬ 
fications/  and  the  Information  Metric  uas  shoutn  to  be  successful 
in  predicting  performance  reliability  data/  but  not  to  be  successful 
in  predicting  error  detection  and  correction  data.  Me  are  noui 
warranted  to  evaluate  the  field  validity  of  the  performance  relia¬ 
bility  function  of  this  model  that  has  been  demonstrated  to  be  valid 
in  the  laboratory.  This  field  evaluation  could  be  accomplished  by 
instrumenting  an  on-line  operational  console  in  a  manner  guided  by 
the  measures  taken  in  the  laboratory  demonstration  that  has  been 
reported. 

In  regard  to  error  detection/  it  was  encouraging  that  detection  was 
positively  correlated  with  performance  reliability.  Correlated  variance 
is  not  random  variance.  Perhaps  further  investigation  of  the  cueing 
events  associated  with  a  broader  range  of  error  events  would  lead 
to  a  basis  for  a  prediction  algorithm.  However/  it  is  not  encouraging 
to  have  found  such  a  low  proportion  of  feedback  detected  errors  com¬ 
pared  to  the  high  proportion  of  "intuitively"  detected  errors.  This 
latter  finding  is  suggestive  of  an  idiosyncratic  phenomenon  rather  than 
a  nomothetic  one. 

On  first  thought  it  would  seem  as  if  the  probability  of  correction 
for  a  task  error  should  be  predicted  by  the  same  variables  that  predict 
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performance  reliability.  If  not  that<  then  one  mould  suspect  that 
correction  probability  should  at  least  be  positively  correlated  with 
performance  reliability.  Neither  of  these  assumptions  was  confirmed 
in  the  present  study.  It  would  appear  that  correction  probability 
may  instead  be  related  to  the  nature  of  events  concomitant  with  and 
following  the  error  process*  such  as  the  error  response  and  the  re¬ 
sponses  that  follow  it.  This  would  suggest  inventorying  the  types 
of  errors  that  are  made  in  specified  situations*  and  attempting  to 
base  a  prediction  algorithm  on  the  relationships  established  among 
such  responses. 
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